Research Perspectives and Challenges for Bitcoin and Cryptocurrencies

Bitcoin has emerged as the most successful cryptographic currency in history. Within two years of its quiet launch in 2009, Bitcoin grew to comprise billions of dollars of economic value, even while the body of published research and security analysis justifying the system’s design was negligible. In the ensuing years, a growing literature has identified hidden-butimportant properties of the system, discovered attacks, proposed promising alternatives, and singled out difficult future challenges. This interest has been complemented by a large and vibrant community of open-source developers who steward the system, while proposing and deploying numerous modifications and extensions. We provide the first systematic exposition of the second generation of cryptocurrencies, including Bitcoin and the many alternatives that have been implemented as alternate protocols or “altcoins.” Drawing from a scattered body of knowledge, we put forward three key components of Bitcoin’s design that can be decoupled, enabling a more insightful analysis of Bitcoin’s properties and its proposed modifications and extensions. We contextualize the literature into five central properties capturing blockchain stability. We map the design space for numerous proposed modification, providing comparative analyses for alternative consensus mechanisms, currency allocation mechanisms, computational puzzles, and key management tools. We focus on anonymity issues in Bitcoin and provide an evaluation framework for analyzing a variety of proposals for enhancing unlinkability. Finally we provide new insights on a what we term disintermediation protocols, which absolve the need for trusted intermediaries in an interesting set of applications. We identify three general disintermediation strategies and provide a detailed comparative cost analysis. I. WHY BITCOIN IS WORTHY OF RESEARCH Consider two opposing viewpoints on Bitcoin in straw-man form. The first is that “Bitcoin works in practice, but not in theory.” At times devoted members of the Bitcoin community espouse this philosophy and criticize the security research community for failing to discover Bitcoin, not immediately recognizing its novelty, and still today dismissing its importance due to a lack of rigorous theoretical foundation. A second viewpoint is that Bitcoin hopelessly relies on an unknown combination of socio-economic factors for its current stability which are intractable to model with sufficient precision, failing to yield a convincing argument for the system’s soundness. Given these difficulties, experienced security researchers may avoid Bitcoin as a topic of study, considering it prudent security engineering to only design systems with precise threat models that admit formal security proofs. We strongly dismiss both of these simplistic approaches and show where each viewpoint fails, forwarding new insights based on multiple examples of existing knowledge. To the first, we contend that while Bitcoin has worked surprisingly well in practice so far, there is an important role for research to play in identifying precisely why this has been possible, moving beyond a blind acceptance of the informal arguments presented with the system’s initial proposal. Furthermore, it is crucial to understand whether Bitcoin will still “work in practice” as practices change. We expect external political and economic factors to evolve, and the system must change if and when transaction volume scales, and the nature of the monetary rewards for Bitcoin miners will change over time as part of the system design. It is not enough to argue that Bitcoin has worked from 2009–2014 and will therefore continue likewise. We do not yet have sufficient understanding to conclude with confidence that Bitcoin will continue to work well in practice, and that is a crucial research challenge that requires insight from computer science theory. To the second viewpoint, we contend that Bitcoin is filling an important niche by providing a virtual currency system without any trusted parties and without pre-assumed identities among the participants. Within these constraints, the general problem of consensus in a distributed system is impossible [6], [89] without further assumptions like Bitcoin’s premise that rational (greedy) behavior can be modeled and incentives can be aligned to ensure secure operation of the consensus algorithm. Yet these constraints matter in practice, both philosophically and technically, and Bitcoin’s approach to consensus within this model is deeply surprising and a fundamental contribution. Bitcoin’s core consensus protocol also has profound implications for many other computer security problems beyond currency1 such as distributed naming, secure timestamping and commitment, generation of public randomness, as well as many financial problems such as self-enforcing (“smart”) contracts, decentralized markets and order books, and distributed autonomous agents. In short, even though Bitcoin is not easy to model, it is worthy of considerable research attention as it may form the basis for practical solutions to exceedingly difficult and important problems. II. OVERVIEW OF BITCOIN A. A Contextualized History We refer the interested reader to existing surveys on the “first wave” of cryptocurrency research [14], [91]. In short, cryptographic currencies date back to Chaum’s proposal for 1As we shall see, it may not be possible to remove the currency functionality and still have a working consensus system. “untraceable payments” in 1983 [25], a system involving bankissued cash in the form of blindly signed coins. Unblinded coins are transferred between users and merchants, and redeemable after the bank verifies they have not been previously redeemed. Blind signatures prevent the bank from linking users to coins, providing unlinkability akin to cash. Throughout the 1990s, many variations and extensions of this scheme were proposed. Significant contributions include: removing the need for the bank to be online at purchase time [26], allowing coins to be divided into smaller units [88] and improving efficiency [24]. Several startup companies including DigiCash [104] and Peppercoin [96] attempted to bring electronic cash protocols into practice but ultimately failed in the market. In fact, no schemes from this “first wave” of cryptocurrency research achieved significant deployment. Moderately hard “proof-of-work” puzzles were proposed in the early 1990s for combatting email spam [38] (although it was never widely deployed for this purpose [66]). Many other applications followed, including proposals for a fair lottery [47], minting coins for micropayments [97], and preventing various forms of denial-of-service and abuse in anonymous networks [9]. The latter, Hashcash, was an alternative to using digital micropayments (e.g., NetBill [107] and Karma [117]). Proof of work was also used to detect sybil nodes in distributed peer-to-peer consensus protocols [6], and is used in Bitcoin consensus for a similar reason. Another essential element of Bitcoin is the public ledger, which makes double-spending detectable. In auditable ecash [102], [103], proposed in the late 1990s, the bank maintains a public database to detect double-spending and ensure the validity of coins, however the notion of publishing the entire set of valid coins was dismissed as impractical (only a Merkle root was published instead). B-money [33], proposed in 1998, appears to be the first system where all transactions are publicly (anonymously) broadcast and stored. Proposed on the Cypherpunks mailing list, b-money received minimal attention from the academic research community. Smart contracts [111], proposed in the early 1990s, enable parties to formally specify an enforceable agreement using cryptography and scripts. This idea portends Bitcoin’s scripting capabilities. In 2008, Bitcoin was announced and a white paper penned under the pseudonym Satoshi Nakamoto was posted to the Cypherpunks mailing list [85], followed quickly by the source code of the original reference client. Bitcoin’s genesis block was mined on or around January 3, 2009.2 The first use of Bitcoin as a currency is thought to be a transaction in May 2010, where one user ordered pizza delivery for another in exchange for 10 000 bitcoins. Since then, increasing number of merchants and services have incorporated Bitcoin in some way, and the price has generally risen, reaching a peak of approximately US$1200 per bitcoin in late 2013. Bitcoin’s history has also been colored by association with 2Famously, the first block contains the string “The Times 03/Jan/2009 Chancellor on brink of second bailout for banks.” crime. Bitcoin was famously used in a black market website, Silk Road [27], which operated from Feb. 2011 until Oct. 2013 when it was seized and shut down by the FBI. Botnets have found Bitcoin mining to be a supplemental source of income [52]. A current US federal court case involves a large Bitcoin-based Ponzi scheme [106]. In 2014, a computer virus called CryptoLocker extorted millions of dollars from victims by encrypting their files and demanding a Bitcoin ransom to release the decryption key [44]. Many users’ Bitcoins have been lost due to theft [37] and collapsed exchanges [82]. B. A Technical Overview We present Bitcoin’s current operation through its three main technical components: transactions (including scripts), the consensus protocol, and the communication network. Bitcoin is exceedingly complex—our goal is to present the system with sufficient technical depth, so the extant literature on Bitcoin, reviewed and evaluated in later sections of this paper, becomes understandable. In particular, a key benefit of our three-component breakdown is that it makes evaluating and systematizing proposed changes (Sections V & VIII) insightful by “decoupling” concepts that may be changed independently. Sources of information on Bitcoin. Bitcoin can be difficult to define as there is no formal specification. The original Bitcoin white paper [85] provides a good overview of Bitcoin’s design philosophy but many important technical details are omitted or out-dated. The reference implementation bitcoind is considered a de facto specification, with further knowledge scattered across a series of “Bitcoin Improvement Proposals” (BIPs), forum postings, online wiki articles, the developer mailing list, and logged IRC discussions.3 We systemize these sources into a precise technical introduction, putting forward the components of the system we consider to be independent design decisions. 1) Transactions & Scripts: The state of the world in Bitcoin is represented by a series of messages called transactions. Among other possibilities, transactions are foremost published to transfer quantities of currency from one user to another. It is important to note that the large (and growing) list of transactions is the only state in Bitcoin. There is no built-in notion of higher-level concepts such as users, account balances or identities—these all exist only to the extent that they can be imputed by analyzing the list of all published transactions. Transaction format. A transaction is an array of inputs and an array of outputs. The entire transaction is hashed using SHA256 and this hash serves as its globally unique transaction ID. Transactions are represented using an ad hoc binary format; this is an early example of an important detail for which bitcoind is the de facto specification. Each output contains an integer value representing a quantity of the Bitcoin currency. The precision of this value 3Which can be found, respectively, at: https://github.com/bitcoin/ bitcoin/bips, https://bitcointalk.org/, https://bitcoin.it/, bitcoin-development@ lists.sourceforge.net, irc://freenode.net/#bitcoin-dev, and irc://freenode.net/ #bitcoin-wizards immediately limits the extent to which units of the currency can be sub-divided; the smallest unit is called a satoshi. By convention, 10 satoshis is considered the primary unit of currency, called one “bitcoin”4 and denoted XBT, BTC or B. Each output also has a short code snippet (in a special scripting language) called the scriptPubKey representing the conditions under which that transaction output can be redeemed, that is, included as an input in a later transaction. Transaction scripts. Typically, the scriptPubKey specifies the hash of an ECDSA public key and a signature validation routine. This script can be redeemed by signing the entire redeeeming transaction using the specified key and is called a “pay-to-pub-key-hash” transaction. The vast majority of Bitcoin transactions are pay-to-pub-key-hash and the system is often described with this being the only possibility, although other transaction types are possible. The scripting language is an ad hoc, non-Turing-complete stack language with fewer than 200 commands called opcodes. They include support for cryptographic operations—e.g., hashing data and verifying signatures. Like the transaction format, the scripting language is only specified by its implementation in bitcoind. Transaction inputs refer to previous transactions by their transaction hash and the index of the output within that transaction’s output array. They must also contain a code snippet which “redeems” that transaction output called the scriptSig. To successfully redeem a previous transaction, the concatenated scriptSig and scriptPubKey must form a program which executes successfully. For pay-to-pub-key-hash transactions, the scriptSig is simply a public key and a signature. Conservation of value. In addition to the requirements that each input of a transaction matches a previous transaction output, and each concatenated script successfully redeems the claimed inputs, transactions are only valid if they satisfy the fundamental constraint that the sum of the values of all transaction outputs is less than or equal to the sum of the values of all inputs. We discuss in Section II-B2 the one exception: the coinbase transaction used to create new units of currency. From transactions to ownership. By themselves, this format of transaction implies several interesting properties. There is no inherent notion of identities or individual accounts which “own” bitcoins. Ownership simply means knowing a private key which is able to make a signature that redeems certain outputs—an individual owns as many bitcoins as they can redeem. Public key hashes, as specified in pay-to-pubkey-hash transactions, effectively function as pseudonymous identities within the system and are referred to as addresses. No linking is required to a user’s real-world name or identifying information. Arguably, there is little that is deeply innovative about Bitcoin’s transaction format. However, the use of a scripting language to specify redemption criteria and the realization that transactions can specify the entire state of the system are non4When capitalized “Bitcoin” refers to the entire system whereas lowercase “bitcoin” refers to one unit of currency. obvious design choices given prior cryptocurrency systems, and both have been standard in essentially all subsequent designs. Some proposals extend the semantics of Bitcoin transactions (often by enhancing the scripting language) without changes to any other components. 2) Consensus and Mining: The need for consensus. A transaction-based currency system would be insecure if currency were transferred by sending transactions between users. While the signatures would limit only the valid recipient of a previous transaction from referencing it in valid follow-up transactions, there is nothing in the transactions themselves to limit Alice from redeeming some transaction input twice in separate transactions sent to Bob and Carol, both of which would appear valid to Bob and Carol. Bitcoin takes a simple approach to solving this double spending attack: all transactions must be published in a global, permanent transaction log and any individual transaction output may only be redeemed in one subsequent transaction. Verifying a transaction now requires verifying the transaction’s scripts as well as ensuring that it is successfully published to the log. In Bitcoin, the log is implemented as a series of blocks of transactions, each containing the hash of the previous block, committing this block as its sole antecedent. It is referred to as the blockchain. Note that this design requires global consensus on the contents of the blockchain. If Bob and Carol see two divergent blockchains, they will still be vulnerable to double-spending attacks. One solution is to use a trusted central authority to collect transactions and publish them in signed blocks. However, this is undesirable as this authority might refuse to publish an individual user’s transactions (effectively freezing their assets), might go offline completely, or might intentionally fork the blockchain to double-spend coins. Nakamoto consensus. Bitcoin instead establishes consensus on the blockchain through a decentralized, pseudonymous protocol dubbed Nakamoto consensus. This can be considered Bitcoin’s core innovation and perhaps the most crucial ingredient to its success. Any party can attempt to add to the chain by collecting a set of valid pending transactions and forming them into a block. The core ingredient is the use of a challenging computational puzzle (usually given the slight misnomer proof of work5) to determine which party’s block will be considered the next block in the chain. The process for choosing a new block is simple: the first announced valid block containing a solution to the computational puzzle is considered correct. Upon hearing it, other participants are meant to turn to finding a followup block. If the found block contains invalid transactions or is otherwise malformed, all other participants are meant to reject this proposed block and continue working until they have found a solution for a valid block. At any given time, the consensus blockchain is the “longest” version. Typically this is simply the branch with the most blocks, but because the mining difficulty 5Bitcoin’s mining puzzle is not a true proof-of-work scheme but a probabilistic one. Finding a solution is computationally challenging on expectation, but it is possible to get lucky and find a solution with very little work. can vary between long forks the longest chain must be defined as the one with the greatest expected difficulty to produce.6 It is also possible for two valid solutions to be found at approximately the same time (depending on network latency), which leads to a temporary fork during which there are two equal-length chains. Miners can choose either fork in this scenario, and due to the random nature of the computational puzzle, one blockchain will eventually be extended further than the other, at which point the miners will shift to it. While the original Bitcoin specification provided only an informal argument that eventual consensus would emerge [85], followup work has proved that, assuming an effective and timely broadcast channel and that miners controlling a majority of computational power follow the protocol faithfully, the protocol is robust and the network gradually reaches consensus [43], [80]. Block confirmation. The gradual nature of this consensus mechanism procedure implies that users must wait for blocks to be found in order to gain high confidence that a transaction is permanently included in the blockchain. During a fork, one of the branches will eventually be orphaned when miners converge on the other. Typically, both branches will include largely the same set of transactions, but if conflicting transactions are included in competing branches then one may be apparently included in the longest branch but then effectively revoked if the other chain branch surpasses it. In the worst case, this will enable the equivalent of a double spending attack [11], [55]. To protect against this risk, users should not consider a transaction to be included until it is in a block which has been “confirmed” by multiple followup blocks. In theory, users can never be completely sure that a transaction won’t eventually be removed by a very deep fork [12]. However when a majority of miners follow the default protocol, users can infer that a transaction is exponentially increasingly likely (see Section III-A) to end up on the eventual longest chain as more confirming blocks are found. In practice, most Bitcoin clients require 6 confirmation blocks before accepting a transaction as “confirmed.” Arbitrary-length forks are also prevented in an ad-hoc manner by including hard-coded blockchain prefixes (checkpoints) with the default Bitcoin client before which clients will not accept a fork. Laurie [65] argues that the existence of these checkpoints means Bitcoin is not in fact a distributed consensus protocol, and without them eventual consensus would not exist because a future majority miner could always re-write history from the genesis block. Incentivizing correct behavior. A critical component of this protocol is that a participant who finds a block is allowed to insert a coinbase transaction minting a specified amount of currency and transferring it to an address of their choosing. Because participants to the consensus protocol are working (indeed, racing) to solve this computational puzzle in exchange 6Specifically, this prevents an attacker from forking the blockchain, modifying timestamps on their fork to produce a lower difficulty, and using this lower difficulty to more easily overtake the previous longest chain. for monetary rewards, they are called miners. The new currency incentivizes miners to only work towards finding valid blocks, as invalid ones will be rejected by the network and their mining rewards will then not exist in the eventuallylongest blockchain. Note that “valid” blocks, from the point of view of miners, are simply blocks which they believe the majority of other miners will accept and build upon, trumping any formal specification of validity (for which there is none beyond the bitcoind implementation). Also note that this consensus algorithm relies on monetary rewards for miners and hence cannot easily be used in systems with no notion of transferable value. In Bitcoin, miners receive all new currency initially and there is no other allowed mechanism for money creation. This is not strictly essential, but the consensus protocol does require some monetary reward is issued to miners or else they have no incentive to find valid blocks and compute the difficult proof-of-work puzzle. Mining details. The computational puzzle itself requires finding a partial pre-image for SHA-256, a cryptographic hash function. Specifically, the puzzle is to find a block (consisting of a list of transactions, the hash of the previous block, plus an arbitrary nonce value) whose SHA-256 hash is less than a target value. The puzzle is often formulated by the following approximation: finding a hash that starts with d consecutive zero bits.7 In so far as the hash output is statistically random, miners can do no better than an exhaustive search over the space of possible nonces for a desired block [9]. The random aspect of this puzzle is important; with a non-randomized proof-of-work function the most powerful individual miner could be expected to find every block first but with a randomized function any miner will have a probability of finding the next block proportional to their share of the competing computational power. The difficulty of the puzzle is calibrated so that a new block is found, on average, once every 10 minutes. To maintain this, the difficulty is adjusted once every 2016 blocks, or approximately every two weeks, by a deterministic function of the timestamps included in each of the previous 2016 blocks. Mining rewards and fees. The amount of currency miners may create in each block through a coinbase transaction (the block reward) is determined by a fixed schedule. Initially, each block created 50 new bitcoins. This has since halved to B25, and is scheduled to halve roughly every four years until 2140 at which point no new bitcoins will be created. To enable this wind-down of currency creation, miners do not only profit from block rewards: they are also allowed to claim the net difference in value between all input and all output transactions in this block. For users, a block with greater input value than output value thus includes a transaction fee paid to the miners in exchange for publishing their transaction. To date, transaction fees have primarily been used to discourage overuse of the network and have never provided more than about 1–2% of mining revenue [83]. Fee values have primarily been determined by defaults configured in the 7At the time of this writing d ≈ 68. reference client [83], with a small number of users opting to pay higher fees in an attempt to have their transactions published more quickly. Mining pools. In practice, miners often collaborate in mining pools [99], although these were not described in the original protocol design and may have been unanticipated. Mining pools are typically administered by a manager who pays miners to mine blocks on their behalf (allocating mining rewards to a key controlled by the pool manager). When blocks are found, the pool manager shares the profits among pool members proportional to the amount of work performed. for example. Participating miners can easily prove (proabilisitically) the amount of work they have performed by sending “near-blocks” whose hash starts with a large number of zeros (say d′ = 40) but not enough to make them valid Bitcoin blocks. Pools allow miners to significantly lower the variance in their mining payout, at the cost of a small fee that is paid to the pool manager which lowers their expected total reward. Since 2013, the majority of mining power has been organized into pools. There are several standard protocols for low-latency communication from pool operators to members [90] and between the operators of different pools [30], [68]. While the most popular pools are centrally administered, many miners form ad hoc pools using the p2pool protocol [118]. 3) Peer-to-Peer Communication Network: The final core component of Bitcoin is its communication network. Essentially, it is a decentralized, ad hoc peer-to-peer broadcast network used to propose new transactions and announced newly-mined blocks. Generally, this is the least innovative of the three components and few alternative proposals have made substantial changes. The performance and stability of the network has an important impact on the consensus protocol for two reasons. First, any latency between the discovery of a block and its receipt by all other nodes increases the possibility of a temporary fork. Fear of frequent forks motivated the choice of 10 minutes as the block creation time in the original design. Second, a malicious miner who is able to control a substantial portion of the network may attempt to favor the broadcast of their own blocks, increasing the likelihood of their blocks “winning” a fork and thus increasing their expected mining rewards. Similarly, any party able to censor the network can selectively block transmissions and freeze assets. Thus it is important for Bitcoin to have a broadcast network which is decentralized (fitting with its overall design), low latency, and where it is difficult to censor or delay messages. Network topology and discovery. Any node can join the network by connecting to a random sample of other nodes. By default, each node attempts to make 8 outgoing connections, and is prepared to receive up to 125 incoming connections. Nodes behind a NAT, such as mobile clients, are unable to receive incoming connections. Peers who join the network initially need a way to find out about other peers. Like many other peer-to-peer networks, Bitcoin achieves this through the use of dedicated directory servers or “seed nodes,” the identities of whom are hard coded into the reference client; thereafter, each node maintains a list of peer addresses it knows about. Peers also propagate information about each other through two other mechanisms: when a node establishes a new outgoing connection, it triggers a cascade of relay messages containing its connection information; second, upon receiving an incoming connection, a node asks its peer for a sample from its list of known-about addresses. Overall, the effect of this mechanism is to establish a well-connected random network, with low degree yet low diameter, suitable for rapid broadcast of information through diffusion [35], [56]. Communication protocol. New blocks and pending transactions are broadcast to the entire network by flooding. Nodes send INV messages to all of their peers containing the hashes of new blocks or pending transactions whenever they first hear of them. Peers can respond by requesting the full contents of these blocks or transactions if they have not yet seen them (via a GETDATA message). Nodes will: only forward new data once, preventing infinite propagation; only relay transactions and blocks that are valid; only relay the first block they hear of when two blocks are found in a temporary fork; and will not broadcast pending transactions which conflict (doublespend) with pending transactions they have sent. These limits are performance optimizations designed to limit data on the network—a non-compliant node may relay invalid or conflicting data, requiring all nodes to independently validate all data they receive. Relay policy. By default, Bitcoin nodes only relay transactions and blocks which satisfy stricter validation rules than what is permitted by the transaction validity rules. The goal is to prevent various denial of service attacks—an application of the classic robustness principle “be conservative in what you send, be liberal in what you accept.” For example, default nodes only relay transactions containing scripts from a very narrow whitelist of standard transaction types. The implication of this policy is that users of the system wishing to have non-standard transactions included in the blockchain cannot use the normal Bitcoin network, but will need to contact an agreeable miner directly.8 Another example is that default nodes refuse to relay more than a few thousand transactions below 0.001 XBT per minute as a “penny-flooding” defense. III. STABILITY OF BITCOIN A key open question regarding Bitcoin is under what conditions the protocol is stable. Stability has been defined in multiple conflicting ways, but it is broadly taken to means that the system will continue to behave in a way that facilitates a functional currency system as it grows and participants attempt novel attacks. We will consider notions of stability for each component of Bitcoin in turn. A. Stability of transaction rules How participants in the Bitcoin ecosystem achieve consensus about the validity rules for Bitcoin transactions is underanalysed. The baseline philosophy is that the rules were set in 8For example, Andrychowicz et al. [5] reported needing to submit their complex multiparty lottery scripts directly to the Eligius mining pool. stone by Satoshi, which we can call canonicalism. This has mediated some disagreements about the specified rules, such as a benign bug in the original OP_CHECKMULTISIG opcode which has been preserved as canonical. However, canonicalism cannot fully explain the current rules of Bitcoin. Several changes to the rules have been implemented to add new features (e.g., pay-to-script-hash [2]). Rules have also been modified to fix bugs, with the best-known example occurring in March 2013 when a bug limiting the size of valid blocks was removed. This caused a fork as new, larger blocks were rejected by unpatched clients. To resolve this, the updated clients abandoned a 24-block fork and temporarily ceased including larger blocks during a two-month window for older clients to upgrade [1]. Eventually however, the bug fix won out and unpatched clients, while arguably implementing a more canonical version of the rules, were excluded. Within the technical rules of Bitcoin, no process is specified for updating or evolving the rules. Without unanimity among miners, any major change may permanently fork the system, with different populations considering the longest blockchain reflecting their interpretation of the rules to be authentic, regardless of its length relative to other blockchains. At this point, it would no longer be clear which version is “Bitcoin.” Thus despite the popular conception of Bitcoin as a fully decentralized system, the need for rule changes (or disambiguation) means some level of governance is inherently required to maintain real-world consensus about what is considered Bitcoin [59]. Currently, de facto governance is provided by the core Bitcoin developers who maintain bitcoind, with the Bitcoin Foundation providing a basic organizational structure and raising a small amount of funding through donations to support the development team. As with many early Internet protocols, there is as of yet no formal process for taking decisions beyond